Dichotomous Thinking: a Problem beyond Nhst

The cliff effect—a sudden drop of confidence that a real effect exists just above p=0.05—captures the way many researchers and students interpret p-values. It is consistent with dichotomous judgements based exclusively on statistical significance (SS). Many have argued that CI can overcome over-reliance on SS. In our study, 172 researchers rated the strength of evidence against the null hypothesis as a function of 8 p-values crossed with 2 sample sizes. A further 86 received the same results presented as CIs. Although the cliff was sometimes found with p-values (23% of 172), it was more frequent with CIs (32% of 86). Thus, the argument that CIs can reduce over-reliance on SS may be overstated. Students, and also researchers, should be trained to think in terms of (or to ask) quantitative (how much A and B differ) rather than dichotomous research questions, whether analysis relies on SS or CIs.